Evaluation of a technical advisory board for an occupational injury surveillance research project: A qualitative study

Abstract Background and Aims Advisory boards play a key role in guiding and informing research programs, including occupational health surveillance. It is important to evaluate the effectiveness of these advisory boards. This report details the organization of the Risk Information System for Commercial (RISC) Fishing Technical Advisory Board (TAB), the approach taken to evaluate the TAB, and the results of the evaluation. The RISC TAB was formed to provide advice and recommendations to the study team and informed the development and use of the safety surveillance system. Methods The evaluation approach was informed by limited previous literature on advisory board assessments. This evaluation was conducted in Year 5 of the 6‐year project. A review of the meeting notes, materials and correspondences, and study progress was conducted internally to document input from the board and associated actions. To obtain member perspectives, we surveyed the TAB and discussed it in a subsequent TAB meeting. Results The RISC Fishing TAB members constitute a wide variety of commercial fishing safety stakeholders. The internal analysis identified the main project aspects and 14 of the proposed changes from the TAB that have either been implemented or are in progress in the project. Ten of the 15 TAB members responded indicating a positive experience on board organization and conduct. Conclusion Evaluation of advisory boards is an essential part of a research program. A process is outlined in this report to inform future efforts to document measurable ways to inform projects based on advisory board feedback and reflections.

Commercial (RISC) Fishing, an OSH surveillance research project. RISC Fishing is a recently developed, comprehensive commercial fishing injury surveillance system that utilizes data from US Coast Guard (USCG) injury reports, state and national Trauma Registries, and Emergency Medical Service (EMS) data to estimate injury risk and risk factors in commercial fishing in the Pacific Northwest (PNW). RISC Fishing aims to provide a resource for PNW commercial fisheries to develop, evaluate, and inform safety initiatives by combining these sources of data into an ongoing, scalable, adaptable surveillance system and to trial the use of the system for hazard assessments and evaluations of safety and fishery management interventions. Previous publications have outlined both the process of matching data 2 and the effort to explore publicly available charts to obtain commercial fishing-specific estimates of workplace exposures, safety behaviors, health status, illnesses, and injuries, as well as working and employment conditions. 3 As noted in the Center for Disease Control's Updated Guidelines for Evaluating Public Health Surveillance Systems, it is important that surveillance systems and advisory boards that oversee them are evaluated regularly. 4 Many private entities, nonprofit organizations, and academic institutions operate advisory boards but little research has been done on the impact of advisory boards and how to evaluate them. A recent effort to improve the evaluation of OSH surveillance systems has been undertaken, 5 but did not specifically address advisory board evaluations. Some literature was available describing an evaluation of community health-based advisory boards and private entity advisory boards. 6,7 We found no published research looking at scientific advisory board members' level of satisfaction or a framework for evaluating the advisory boards they serve on.
The RISC Fishing TAB (RISC TAB) was established in 2017 and has met biannually throughout the project period (5 years to date).
This report details the operation of the RISC TAB, the approach taken to evaluate the TAB, and the results of the evaluation. Our report can help to improve the utilization of future advisory boards that will benefit OSH research projects.

| METHODS
The purpose of the RISC TAB is to identify relevant commercial fishing industry issues and priorities and to help translate surveillance findings into tailored, nimble responses for hazard assessment needs of important fishery safety stakeholders. The RISC TAB provides advice and recommendations to the study team and informed the development and use of the safety surveillance system. Projectspecific topics shared with the TAB included the: (1) database architecture; (2) data elements and coding; (3) data entry and query interfaces; (4) reporting elements; (5) report contents and data visualization; (6) priorities for hazard assessments; and (7) dissemination materials and methods (e.g., tutorials, hazard sheets).
RISC TAB members were recruited based on their interest, ability to represent key stakeholders, and availability to commit time to the project. As this project is currently restricted to Oregon and Washington, all members represented commercial fishing interests in those states. The RISC TAB members constitute a wide variety of commercial fishing safety stakeholders (including state and national government agencies), public health surveillance practitioners, and OSH researchers and practitioners (Table 1).
To date, nine semiannual meetings have been held over the 5 years of the project with meetings occurring both virtually and in person. During the pandemic, all meetings were virtual, but because the TAB was already well established, it continued to function effectively. Meetings were conducted in a professional but comfortable manner and facilitated by the project Principal Investigators. At the onset, TAB expectations were set with appropriate ground rules (see Table 2). 8 Agendas, including specific premeeting actions for board members, were shared ahead of time, and expectations were set with appropriate ground rules, and meeting notes/decisions/ action items circulated regularly. Approval of activities by an Institutional Review Board was determined to not be necessary due.

| RISC TAB EVALUATION
To determine the TAB's effectiveness and impact, we analyzed The initial draft of the survey contained a section on a self-evaluation which was removed to keep the survey brief and most relevant. Table 3 shows the final survey.

| RESULTS
At the time of the evaluation, the RISC TAB had 13 of the original 16 members serving continuously since 2017 with a retention rate of 83%. All current board members have been attending meetings for at least the last 3 years.
T A B L E 2 RISC Fishing TAB ground rules 1. Stick to agenda topics.

2.
Keep the discussion focused on one subject at a time.
3. Discuss all relevant information and issues, even difficult ones.

4.
Keep the discussion open and balanced.

5.
Speak for yourself, not for others in the room.
8. Be respectful of others.
9. Disagree openly, but try not to be disagreeable.

10.
Look for mutually beneficial solutions.
11. Contribute to the discussion.
12. Follow through on commitments.
Abbreviations: RISC, Risk Information System for Commercial; TAB, Technical Advisory Board.

T A B L E 3 RISC Fishing Advisory Board Survey
Likert scale questions (5-point scale: Strongly agree to Strongly disagree) 1. Advisory board members are knowledgeable about commercial fishing needs and the issues that the industry face.
2. Advisory board meetings are productive.
-If you disagree, please share with us any concerns and ideas you have.

3.
There are a sufficient number of advisory board meetings throughout the year.
-If not, how many do you recommend?
4. The quality, quantity, and timing of the information given to advisory board members is adequate.
-If you disagree, please share with us any concerns and ideas you have.

5.
The agendas of our meetings and supporting written material are provided in advance of meetings.
6. Board meetings are generally well-run and make good use of members' time.
-If you disagree, please share with us any concerns and ideas you have.

7.
Our board's size is about right.
The open-ended questions generated thoughtful responses. • what key drivers of risk-taking such as weather, bar closings, season-opening, and season timing could have to alleviate risk, • sleep deprivation, • hearing loss, • boatyard hazards, • other fishermen's health issues, • safety management systems for small fishing vessels.

ACKNOWLEDGMENTS
The authors would like to acknowledge our colleagues at the Pacific Northwest Agricultural Safety and Health Center for reviewing the methods and survey. We would also like to thank all of our advisory board members for their time and expertise. This study was funded by the National Institute of Occupational Health and Safety; Grant number: #2U54OH007544-16. NIOSH had no role in the design of the study and collection, analysis, and interpretation of data and in the writing of the manuscript.

CONFLICT OF INTEREST
The authors declare no conflict of interest.

DATA AVAILABILITY STATEMENT
The data sets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Amelia Vaughan accepts full responsibility for the accuracy and integrity of the data provided.

TRANSPARENCY STATEMENT
This manuscript is an honest, accurate, and transparent account of the study being reported; no important aspects of the study have been omitted, and any discrepancies from the study as planned (and, if relevant, registered) have been explained.